Method and system for calculating importance of a block within a display page

ABSTRACT

A method and system for identifying the importance of information areas of a display page. An importance system identifies information areas or blocks of a web page. A block of a web page represents an area of the web page that appears to relate to a similar topic. The importance system provides the characteristics or features of a block to an importance function that generates an indication of the importance of that block to its web page. The importance system “learns” the importance function by generating a model based on the features of blocks and the user-specified importance of those blocks. To learn the importance function, the importance system asks users to provide an indication of the importance of blocks of web pages in a collection of web pages.

TECHNICAL FIELD

The described technology relates generally to identifying a block withina display page that represents the primary topic of the display page.

BACKGROUND

Many search engine services, such as Google and Overture, provide forsearching for information that is accessible via the Internet. Thesesearch engine services allow users to search for display pages, such asweb pages, that may be of interest to users. After a user submits asearch request that includes search terms, the search engine serviceidentifies web pages that may be related to those search terms. Toquickly identify related web pages, the search engine services maymaintain a mapping of keywords to web pages. This mapping may begenerated by “crawling” the web (i.e., the World Wide Web) to identifythe keywords of each web page. To crawl the web, a search engine servicemay use a list of root web pages to identify all web pages that areaccessible through those root web pages. The keywords of any particularweb page can be identified using various well-known informationretrieval techniques, such as identifying the words of a headline, thewords supplied in the metadata of the web page, the words that arehighlighted, and so on. The search engine service then ranks the webpages of the search result based on the closeness of each match, webpage popularity (e.g., Google's PageRank), and so on. The search engineservice may also generate a relevance score to indicate how relevant theinformation of the web page may be to the search request. The searchengine service then displays to the user links to those web pages in anorder that is based on their rankings.

Whether the web pages of a search result are of interest to a userdepends, in large part, on how well the keywords identified by thesearch engine service represent the primary topic of a web page. Becausea web page may contain many different types of information, it may bedifficult to discern the primary topic of a web page. For example, manyweb pages contain advertisements that are unrelated to the primary topicof the web page. A web page from a news web site may contain an articlerelating to an international political event and may contain “noiseinformation” such as an advertisement for a popular diet, an arearelated to legal notices, and a navigation bar. It has beentraditionally very difficult for a search engine service to identifywhat information on a web page is noise information and what informationrelates to the primary topic of the web page. As a result, a searchengine service may select keywords based on noise information, ratherthan the primary topic of the web page. For example, a search engineservice may map a web page that contains a diet advertisement to thekeyword “diet,” even though the primary topic of the web page relates toan international political event. When a user then submits a searchrequest that includes the search term “diet,” the search engine servicemay return the web page that contains the diet advertisement, which isunlikely to be of interest to the user.

It would be desirable to have a technique for calculating the importanceof various information areas of a web page to the primary topic of theweb page.

SUMMARY

An importance system identifies the importance of information areas of adisplay page. The importance system identifies information areas orblocks of a web page that represent areas of the web page that appear torelate to a similar topic. After identifying the blocks of a web page,the importance system provides the characteristics or features of ablock to an importance function which generates an indication of theimportance of that block to its web page. The importance system maylearn the importance function by generating a model based on thefeatures of blocks and the user-specified importance of those blocks.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram that illustrates the components of theimportance system in one embodiment.

FIG. 2 is a flow diagram that illustrates the processing of a generateimportance function component of the importance system in oneembodiment.

FIG. 3 is a flow diagram that illustrates the processing of a generatefeature vector component of the importance system in one embodiment.

FIG. 4 is a flow diagram that illustrates the processing of a calculateblock importance component of the importance system in one embodiment.

FIG. 5 is a flow diagram that illustrates the processing of an ordersearch result component that uses the learned importance function in oneembodiment.

FIG. 6 is a flow diagram that illustrates the processing of an expandsearch result component that uses the learned importance function in oneembodiment.

FIG. 7 is a flow diagram that illustrates the processing of a retrieveweb page component that uses the learned importance function in oneembodiment.

FIG. 8 is a flow diagram that illustrates the processing of a classifyweb page component that uses the learned importance function in oneembodiment.

DETAILED DESCRIPTION

A method and system for identifying the importance of information areasof a display page is provided. In one embodiment, an importance systemidentifies information areas or blocks of a web page. A block of a webpage represents an area of the web page that appears to relate to asimilar topic. For example, a news article of a web page may representone block, and a diet advertisement of the web page may representanother block. After identifying the blocks of a web page, theimportance system provides the characteristics or features of a block toan importance function that generates an indication of the importance ofthat block to its web page. The importance system “learns” theimportance function by generating a model based on the features ofblocks and the user-specified importance of those blocks. To learn theimportance function, the importance system asks users to provide anindication of the importance of blocks of web pages in a collection ofweb pages. For example, one user may identify a certain block as ofprimary importance to a web page, and another user may identify thatsame block as of secondary importance to the web page. The importancesystem also identifies a feature vector for each block that mayrepresent various spatial and content features of the block. Forexample, a spatial feature may be the size of a block or the location ofa block, and a content feature may be the number of links within a blockor the number of words within the block. The importance system thenlearns an importance function, based on the user-specified importanceand the feature vectors of the blocks, that inputs a feature vector of ablock and outputs the importance of the block. The importance system mayattempt to learn a function that minimizes the square of the differencesbetween the user-specified importance of the blocks and the importancecalculated by the importance function. In this way, the importancesystem can identify a block of a web page that may be related to theprimary topic of the web page based on features of blocks that usersthink are important.

The importance function, once learned, can be used to improve theaccuracy of a wide range of applications whose results depend ondiscerning topics of web pages. A search engine service may use theimportance function to calculate relevance of web pages of a searchresult to the search request. After the search result is obtained, thesearch engine service may use the importance function to identify themost important block of each web page and calculate a relevance of themost important block (or group of importance blocks) to the searchrequest. Because this relevance is based on the information of the mostimportant block of the web page, it may be a more accurate measure ofrelevance than a relevance that is based on the overall information ofthe web page, which may include noise information.

A search engine service may also use the importance function to identifyweb pages that are related to a certain web page. The search engineservice may use the importance function to identify the most importantblock of that web page and then formulate a search request based on thetext of the identified block. The search engine service may perform asearch using that formulated search request to identify web pages thatare related to that certain web page.

A search engine service may also use the importance function to classifyweb pages. For example, web pages may be classified based on theirprimary topic. The search engine service may use the importance functionto identify the most important block of a web page and then classify theweb page based on the text of the identified block.

A web browser may also use the importance function to identify portionsof a web page to be displayed when the entire content of the web pagecannot fit on a display device. For example, devices such as cell phonesor personal digital assistants may have very small displays on whichmost web pages cannot reasonably be displayed because of the amount,size, and complexity of the information of the web page. A web page isnot reasonably displayed when the content is so small that it cannot beeffectively viewed by a person. When such a device is used to browse webpages, the importance function may be used to identify a most importantblock of a web page and to display only the identified block on thesmall display. Alternatively, a web browser may rearrange the blocks ofa web page, highlight blocks of a web page, remove blocks of a web page,and so on based on the importance of the blocks as indicated by theimportance function.

The importance system can identify the information areas of a web pageusing various segmentation techniques such as a document object modelbased (“DOM-based”) segmentation algorithm, a location-basedsegmentation algorithm, a vision-based segmentation algorithm, and soon. A DOM-based segmentation algorithm may use the HTML hierarchy of aweb page to identify its various blocks. A location-based segmentationalgorithm attempts to identify areas of a web page that may beconsidered a unit based on physical characteristics of the areas. Avision-based segmentation algorithm is described in U.S. patentapplication Ser. No. 10/628,766, entitled “Vision-Based DocumentSegmentation,” filed on Jul. 28, 2003, which is hereby incorporated byreference. That vision-based segmentation algorithm identifies blocksbased on the coherency of the information content of each block. Forexample, an advertisement for a diet may represent a single blockbecause its content is a directed to a single topic and thus has a highcoherency. An area of a display page that includes many different topicsmay be divided into many different blocks.

The importance system may use various techniques to learn the importancefunction from the feature vectors representing the blocks and theuser-specified importance of the blocks. For example, as described belowin more detail, the importance system may learn the importance functionusing a neural network or a support vector machine. In addition, theimportance system may use various spatial and content features of ablock in a feature vector representation of the block. For example, thespatial features may include the center location of the block and thedimensions of the block, which are referred to as “absolute spatialfeatures.” These absolute spatial features may be normalized based onthe size of the web page, which are referred to as “relative spatialfeatures,” or normalized based on the size of the window, which arereferred to as “window spatial features.” The content features mayrelate to the images of the block (e.g., number and size of images),links of the block (e.g., the number of links and the number of words ineach link), text of the block (e.g., number of words in block), userinteraction of the block (e.g., number and size of input fields), andforms of the block (e.g., number and size). Various content features mayalso be normalized. For example, the number of links of a block may benormalized by the total number of links on the web page, or the numberof words of text in the block may be normalized by the total number ofwords in the text of all the blocks of the web page.

In one embodiment, the importance system may allow users to specify theimportance of blocks using discrete or continuous values. Higher valuesmay represent a higher importance of a block to a web page. When usingdiscrete values of 1-4, a value of 1 may represent noise informationsuch as an advertisement, a copyright notice, a decoration, and so on. Avalue of 2 may represent useful information that is not particularlyrelevant to the primary topic of the page such as navigationinformation, directory information, and so on. A value of 3 mayrepresent information that is relevant to the primary topic of the pagebut not of prominent importance such as related topics, topic indexes,and so on. A value of 4 may indicate the most prominent part of the webpages such as a headline, main content, and so on.

FIG. 1 is a block diagram that illustrates the components of theimportance system in one embodiment. The importance system may includethe components 101-105 and data stores 111-114. The components mayinclude an identify blocks component 101, a collect user-specifiedimportance component 102, a generate importance function component 103,an apply importance function component 104, and a generate featurevector component 105. The data stores may include a web page store 111,a block store 112, a user-specified importance store 113, and animportance function store 114. The web page store may contain acollection of web pages for use in generating or learning the importancefunction. The identify blocks component identifies the blocks of eachweb page of the web page store and stores the identification of eachblock in the block store. The collect user-specified importancecomponent displays each web page of the web page store to a userhighlighting each of the blocks of the web page, asks the user tospecify the importance of each block, and stores the specifications ofimportance in the user-specified importance store. The generateimportance function component uses the generate feature vector componentto generate a feature vector for each block of the block store. Thegenerate importance function then learns an importance function to modelthe feature vectors of the blocks and the user-specified importance ofthe blocks. The importance function store may contain information suchas factors and parameters that define the learned importance functionthat are calculated by the generate importance function component. Theapply importance function component is passed a block of a web page,invokes the generate feature vector component to generate the featurevector for the block, and applies the importance function to thegenerated feature vector to generate an indication of the importance ofthe block.

The computing device on which the importance system can be implementedmay include a central processing unit, memory, input devices (e.g.,keyboard and pointing devices), output devices (e.g., display devices),and storage devices (e.g., disk drives). The memory and storage devicesare computer-readable media that may contain instructions that implementthe importance system. In addition, the data structures and messagestructures may be stored or transmitted via a data transmission medium,such as a signal on a communications link. Various communications linksmay be used, such as the Internet, a local area network, a wide areanetwork, or a point-to-point dial-up connection.

FIG. 1 illustrates an example of a suitable operating environment inwhich the importance system may be implemented. The operatingenvironment is only one example of a suitable operating environment andis not intended to suggest any limitation as to the scope of use orfunctionality of the importance system. Other well-known computingsystems, environments, and configurations that may be suitable for useinclude personal computers, server computers, hand-held or laptopdevices, multiprocessor systems, microprocessor-based systems,programmable consumer electronics, network PCs, minicomputers, mainframecomputers, distributed computing environments that include any of theabove systems or devices, and the like.

The importance system may be described in the general context ofcomputer-executable instructions, such as program modules, executed byone or more computers or other devices. Generally, program modulesinclude routines, programs, objects, components, data structures, and soon that perform particular tasks or implement particular abstract datatypes. Typically, the functionality of the program modules may becombined or distributed as desired in various embodiments.

In one embodiment, the importance system learns a block importancefunction by minimizing a least squares objective function based onuser-specified importance of blocks of a collection of web pages. Eachblock can be represented by a feature vector and importance pair (x, y)where x is the feature vector of the block and y is its importance. Theset of feature vector and importance pairs of the collection of webpages is referred to as a training set T. The importance system attemptsto learn an importance function ƒ that minimizes the following objectivefunction:Σ_((x,y)∈T)|ƒ(x)−y| ²  (1)The objective function can be solved using classification techniquessuch as a support vector machine if y is discrete and regressiontechniques such as a neural network if y is continuous.

When the importance is represented by continuous real numbers, theimportance system may apply a neural network learning for learning theoptimal ƒ* which is given by minimizing the following objectivefunction:

$\begin{matrix}{f^{*} = {\arg\mspace{14mu}{\min\limits_{f}{\sum\limits_{i = 1}^{m}{{{f\left( x_{i} \right)} - y_{i}}}^{2}}}}} & (2)\end{matrix}$where m is the number of blocks in the training set. This is amultivariate non-parametric regression problem, since there is no apriori knowledge about the form of the true regression function that isbeing estimated.

A neural network model has three major components: architecture, costfunction, and search algorithm. The architecture defines the functionalform relating the inputs to the outputs (in terms of network topology,unit connectivity, and activation functions). The search in weight spacefor a set of weights that minimizes the objective function is thetraining process. In one embodiment, the importance system uses a radialbasis function (“RBF”) network and a standard gradient descent as thesearch technique.

The importance system constructs an RBF network with three layers havingdifferent roles. The input layer comprises source nodes (i.e., sensoryunits) that connect the network to its environment (i.e., low-levelfeature space). The hidden layer applies a nonlinear transformation fromthe input space to the hidden space. Generally, the hidden space is ofhigh dimensionality. The hidden layer has RBF neurons, which calculatethe input of the hidden layer by combining weighted inputs and biases.The output layer is linear and supplies the block importance given thelow-level block representation applied to the input layer.

The function learned by RBF networks can be represented by thefollowing:

$\begin{matrix}{{f_{i}(x)} = {\sum\limits_{j = 1}^{h}{\omega_{ij}{G_{i}(x)}}}} & (3)\end{matrix}$where i is a block in the training set, h is the number of hidden layerneurons, ω _(ij)∈ R are the weights, and G_(i) is the radial functiondefined as follows:

$\begin{matrix}{{G_{i}(x)} = {\exp\left( {- \frac{{{x - c_{i}}}^{2}}{\sigma_{i}^{2}}} \right)}} & (4)\end{matrix}$where c_(i) is the center for G_(i) and σ_(i) is the basis functionwidth. The k-dimensional mapping can be represented as follows:x→ƒ(x)=(ƒ₁(x), ƒ₂(x), . . . , ƒ_(k)(x))  (5)where ƒ=[ƒ₁, ƒ₂, . . . , ƒ_(k)] is the mapping function.

The RBF neural network approximates the optimal regression function fromfeature space to block importance. The importance system can train theRBF neural network off-line with the training samples {x_(i), y_(i)}(i=1, . . . , m) of the training set T. For a new block previouslyunprocessed, the importance system can calculate its importance usingthe regression function ƒ given the feature vector of the block.

When the importance is represented by discrete numbers, the importancesystem applies a support vector machine for learning the importancefunction. A support vector machine attempts to minimize structural risk,rather than empirical risk. The support vector machine may use a binaryclassification such that the training set is defined by the following:D={x _(i) , y _(i)}_(i=1) ^(t)  (6)where D is the training set, t is the number of training samples, andy_(i)∈{−1,1}. The importance system attempts to select, among theinfinite number of linear classifiers that separate the data, the onewith the minimum generalization error. A hyperplane with this propertyis the one that leaves the maximum margin between the two classes. Theimportance function may be represented by the following:

$\begin{matrix}{{f(x)} = {{sign}\left( {{\sum\limits_{i = 1}^{t}{\alpha_{i}y_{i}\left\langle {x_{i},x} \right\rangle}} - b} \right)}} & (7)\end{matrix}$where α_(i) associated with the training sample x_(i) expresses thestrength with which that point is embedded in the final function and bis the intercept also known as the bias in machine learning. A propertyof this representation is that often only a subset of the points will beassociated with non-zero α_(i). These points are called support vectorsand are the points that lie closest to the separating hyperplane. Thenonlinear support vector machine maps the input variable into a highdimensional (often infinite dimensional) space, and applies the linearsupport vector machine in the space. Computationally, this can beachieved by the application of a (reproducing) kernel. The correspondingnonlinear decision function is represented as follows:

$\begin{matrix}{{f(x)} = {{sign}\left( {{\sum\limits_{i = 1}^{t}{\alpha_{i}y_{i}{K\left( {x_{i},x} \right)}}} - b} \right)}} & (8)\end{matrix}$where K is the kernel function. Some typical kernel functions include apolynomial kernel, Gaussian RBF kernel, and sigmoid kernel. For amulti-class classification problem, the importance system can apply aone-against-all scheme.

FIG. 2 is a flow diagram that illustrates the processing of a generateimportance function component of the importance system in oneembodiment. In block 201, the component retrieves the collection of webpages from the web page store. In block 202, the component identifiesthe blocks within the retrieved web pages and stores the indications ofthe blocks in the block store. In block 203, the component collectsuser-specified importance data for the blocks. The component may displayeach web page along with an indication of the blocks of the web page andask a user to rate the importance of each block to the web page. Thecomponent stores the user-specified importance in the user-specifiedimportance store. In block 204, the component generates a feature vectorfor each block and may store the feature vectors in the block store. Inblock 205, the component learns the importance function using a neuralnetwork or a support vector machine technique as described above.

FIG. 3 is a flow diagram that illustrates the processing of a generatefeature vector component of the importance system in one embodiment. Thecomponent is passed an indication of a web page along with an indicationof a block of the web page whose importance is to be calculated. Inblock 301, the component identifies the absolute spatial features of theblock. In block 302, the component calculates the window spatialfeatures of the block. The component may calculate the window spatialfeatures such as block center for the y-coordinate according to thefollowing:

$\begin{matrix}{{BlockCenterY} = \left\{ \mspace{11mu}\begin{matrix}{\;{{{BlockCenterY}/\left( {2*{HeaderHeight}} \right)};}} \\{\;\begin{matrix}{{{{if}\mspace{14mu}{BlockCenterY}} < {{HeaderHeight}\mspace{14mu} 0.5}};} \\{{{if}\mspace{14mu}{HeaderHeight}} < {BlockCenterY} <} \\{\mspace{25mu}{{PageHeight} - {{FooterHeight}\mspace{14mu} 1} -}} \\\begin{matrix}{\mspace{25mu}{\left( {{PageHeight} - {BlockCenterY}} \right)/}} \\{\left( {2*{FooterHeight}} \right);} \\{otherwise}\end{matrix}\end{matrix}}\end{matrix} \right.} & (9)\end{matrix}$where HeaderHeight and FooterHeight are predefined constant valuesrelating to the heights of headers and footers of a page. In block 303,the component identifies the content features of the block. In block304, the component normalizes the content features as appropriate andthen returns the feature vector.

FIG. 4 is a flow diagram that illustrates the processing of a calculateblock importance component of the importance system in one embodiment.The component is passed an indication of a web page and calculates theimportance of each block of the web page by applying the learnedimportance function. In block 401, the component identifies the blocksof the passed web page. In blocks 402-406, the component loopscalculating the importance of each block of the passed web page. Inblock 402, the component selects the next block of the passed web page.In decision block 403, if all the blocks of the passed web page havealready been selected, then the component returns the importance of eachblock, else the component continues at block 404. In block 404, thecomponent invokes the generate feature vector component to generate thefeature vector for the selected block. In block 405, the componentapplies the learned importance function to the generated feature vectorto calculate the importance of the selected block. In block 406, thecomponent saves the calculated importance and then loops to block 402 toselect the next block of the passed web page.

FIGS. 5-8 illustrate the use of the learned importance function invarious applications. FIG. 5 is a flow diagram that illustrates theprocessing of an order search result component that uses the learnedimportance function in one embodiment. The component is passed a searchresult of web pages and returns the search result reordered based onblock importance as calculated by the learned importance function. Inblock 501, the component selects the next web page of the search result.In decision block 502, if all the web pages of the search result havealready been selected, then the component continues at block 505, elsethe component continues at block 503. In block 503, the componentinvokes a calculate block importance component to calculate theimportance of each block of the selected web page. In block 504, thecomponent sets the relevance of the web page based on the block with thehighest importance. The component may calculate the relevance based onhow well the text of the most important block (or blocks) matches thesearch request for which the search result was identified. The componentmay also combine that relevance with the relevance of each web page thatwas calculated by a search engine. The component then loops to block 501to select the next web page of the search result. In block 505, thecomponent sorts the web pages based on the set relevance and thenreturns the reordered web pages of the search result.

FIG. 6 is a flow diagram that illustrates the processing of an expandsearch result component that uses the learned importance function in oneembodiment. The component is passed a web page and identifies web pagesthat may be related to the passed web page. In block 601, the componentinvokes the calculate block importance component to calculate theimportance of each block of the passed web page. In block 602, thecomponent selects the block of the passed web page with the highestimportance. In block 603, the component retrieves the text associatedwith the selected block. In block 604, the component formulates a searchrequest based on the retrieved text. In block 605, the component submitsthe formulated search request to a search engine service and receivesthe search result in return. The component then returns the searchresult.

FIG. 7 is a flow diagram that illustrates the processing of a displayweb page component that uses the learned importance function in oneembodiment. This component is passed a uniform resource locator of a webpage and displays the content of the block of that web page with thehighest importance as the content of the web page itself. In block 701,the component uses the passed uniform resource locator to retrieve theweb page. In block 702, the component invokes the calculate blockimportance function to calculate the importance of each block of theretrieved web page. In block 703, the component selects the block of theretrieved web page with the highest importance. In block 704, thecomponent displays the selected block as the content of the web page.

FIG. 8 is a flow diagram that illustrates the processing of a classifyweb page component that uses the learned importance function in oneembodiment. The component is passed a web page and classifies that webpage. In block 801, the component invokes the block importance functionto calculate the importance of each block of the passed web page. Inblock 802, the component selects the block (or group of blocks) of thepassed web page with the highest importance. In block 803, the componentretrieves the text of the selected block with the highest importance. Inblock 804, the component generates a classification based on theretrieved text. The classification may also be based on links within theselected block. The component then returns the classification.

One skilled in the art will appreciate that although specificembodiments of the importance system have been described herein forpurposes of illustration, various modifications may be made withoutdeviating from the spirit and scope of the invention. The principles ofthe importance system may be used to determine the importance ofinformation areas within various information sources other than webpages. These information sources may include documents represented usingHTML, XML, or other markup languages. One skilled in the art willappreciate that some applications of block importance may use animportance function that is not “learned.” For example, a developer cansimply define an importance function that reflects their concept ofblock importance and not rely on collecting empirical data of userratings of block importance. Such an importance function could rate theimportance of a block based on a combination of its size and location.Accordingly, the invention is not limited except by the appended claims.

1. A method in a computer system for identifying importance of blocks toa primary topic of a page that contains the blocks, the methodcomprising: providing a collection of training pages with blocks; foreach block of a training page with non-noise information, generating afeature vector that represents the block; and collecting from a personan indication of relative importance of the block to the primary topicof its page; learning a function from the generated feature vector andthe indicated importance of each block that represents a mapping fromfeature vector to importance; and applying the learned function to afeature vector representation of the blocks of the page to generate anindication of the relative importance of each block to a primary topicof the page.
 2. The method of claim 1 wherein the feature vectorincludes spatial features.
 3. The method of claim 1 wherein the featurevector includes content features.
 4. The method of claim 3 wherein thecontent features include image features.
 5. The method of claim 3wherein the content features include link features.
 6. The method ofclaim 3 wherein the content features include interaction features. 7.The method of claim 1 wherein the feature vector includes spatialfeatures and content features.
 8. The method of claim 1 wherein thelearning of the function includes applying a neural network-basedregression analysis.
 9. The method of claim 8 wherein the neural networkis a radial basis neural network.
 10. The method of claim 1 wherein thefunction is learned by selecting a function that tends to minimize thesum of the square of the differences of the importance calculated by thefunction and the collected importance.
 11. The method of claim 1 whereinthe learning of the function includes using a support vector machinealgorithm.
 12. A computer-readable medium containing instructions forcausing a computer system to determine importance of blocks of a page bya method comprising: identifying blocks of a page; and for each block ofthe page, generating a feature vector representing the block; andapplying an importance function to the feature vector to determine theimportance of the block to the page, the importance function having beenlearned using training data comprising feature vectors of blocks andimportance values collected from users.
 13. The computer-readable mediumof claim 12 wherein the importance function is learned using a neuralnetwork.
 14. The computer-readable medium of claim 12 wherein theimportance function is learned using a support vector machine.
 15. Thecomputer-readable medium of claim 12 wherein the feature vector includesspatial features.
 16. The computer-readable medium of claim 12 whereinthe feature vector includes content features.
 17. A computer system forgenerating an importance function for identifying importance of a blockto a primary topic of a page that contains the block, comprising: astore containing a collection of pages with blocks; a component thatgenerates training data by generating feature vectors that representblocks of the collection and collects indications of the importance ofthe blocks to the primary topic of their pages; and a component thatlearns the importance function from the generated training datacomprising the generated feature vectors and the indicated importancesof the blocks, the importance function representing a mapping fromfeature vectors to relative importance of the blocks to the primarytopic of their pages.
 18. The computer system of claim 17 including acomponent that applies the learned importance function to a featurevector representation of a block of a page that is not in the collectionof pages used to create training data, to generate an indication of theimportance of the block to the page.
 19. The computer system of claim 17wherein the feature vector includes spatial features.
 20. The computersystem of claim 17 wherein the feature vector includes content features.21. The computer system of claim 20 wherein the content features includeimage features.
 22. The computer system of claim 20 wherein the contentfeatures include link features.
 23. The computer system of claim 20wherein the content features include interaction features.
 24. Thecomputer system of claim 17 wherein the feature vector includes spatialfeatures and content features.
 25. The computer system of claim 17wherein the learning of the importance function includes applying aneural network-based regression analysis.
 26. The computer system ofclaim 25 wherein the neural network is a radial basis neural network.27. The computer system of claim 17 wherein the importance function islearned by selecting based on minimizing the sum of the square of thedifferences of the importances calculated by the importance function andthe collected importances.
 28. The computer system of claim 17 whereinthe learning of the importance function includes using a support vectormachine algorithm.
 29. The computer system of claim 17 wherein the pagesare web pages and including: a component that determines importance ofblocks of a web page to the primary topic of the web page using thelearned importance function.
 30. The computer system of claim 29including: a component that determines relevance of the web page to asearch request based on relevance of the block of the web page that isdetermined to be most important to the search request.
 31. The computersystem of claim 30 including: a component that ranks web pages based onthe determined relevance of the web pages.
 32. The computer system ofclaim 31 including: a component that displays an indication of the webpages in an order based on their ranking.
 33. The computer system ofclaim 29 including: a component that formulates a search request basedon text of the block of a web page that is determined to be mostimportant and performs a search for web pages using the formulatedsearch request.
 34. The computer system of claim 29 including: acomponent that classifies a web page based on text of the block of theweb page that is determined to be most important.
 35. The computersystem of claim 29 including: a web browser component that displays onlythe block of a web page that is determined to be most important.
 36. Thecomputer system of claim 29 including: a web browser component thatrearranges blocks of a web page based on importance of the blocks anddisplays the rearranged blocks.
 37. The computer system of claim 29including: a web browser component that highlights the block of a webpage that is determined to be most important when displaying the webpage.
 38. The computer system of claim 29 including: a web browsercomponent that removes blocks of a web page that are not the blocksdetermined to be most important.
 39. The method of claim 1 wherein thepages are web pages and including: calculating importance of blocks of aweb page to the primary topic of the web page using the learnedimportance function; and calculating relevance of the web page to asearch request based on relevance of the most important block of the webpage to the search request.
 40. The method of claim 38 including:ranking web pages based on the calculated relevance of the web pages;and displaying an indication of the web pages in an order based on theirranking.
 41. The method of claim 1 wherein the pages are web pages andincluding: formulating a search request based on text of the mostimportant block of a web page.
 42. The method of claim 41 including:performing a search for web pages based on the formulated searchrequest.
 43. The method of claim 1 wherein the pages are web pages andincluding: classifying a web page based on text of the most importantblock of the web page.
 44. A computer system for ranking web pages,comprising: a component that receives a search request; a component thatsearches for web pages that are relevant to the search request, therelevance of a web page to a search request being based on the relevanceof its most important block to a primary topic of the web page, theimportance of a block determined using a importance function that islearned using training data; and a component that displays indicationsof web pages that are relevant to the search request in an order basedon their relevance.
 45. The computer system of claim 44 including: astore containing a collection of web pages with blocks; a component thatgenerates the training data by generating feature vectors that representblocks of the collection and collecting indications of the importance ofthe blocks to the primary topics of their pages; and a component thatlearns the importance function from the generated training data, theimportance function representing a mapping from feature vector toimportance.
 46. The computer system of claim 44 including: a componentthat, when requested to display a web page that is relevant to thesearch request, displays only the most important block of the web page.47. The computer system of claim 44 including: a component that, whenrequested to display a web page that is relevant to the search request,displays blocks of the web page in an order based on importance of theblocks.
 48. The computer system of claim 44 including a component that,when requested to display a web page that is relevant to the searchrequest, displays blocks of the web page in an order based on importanceof the blocks.
 49. The computer system of claim 44 including: acomponent that, when requested to display a web page that is relevant tothe search request, displays blocks of the web page highlighting themost important block of the web.
 50. The computer system of claim 44including: a component that formulates a search request based on text ofthe most important block of a web page that is relevant to the receivedsearch request and performs a search for web pages that are relevant tothe formulated search request, the relevance of a web page to theformulated search request being based on the relevance of its mostimportant block to a primary topic of the web page.